STINFO  COPY 

AFRL-HE-WP-TP-2 007-00 13 

Collaboration  Technologies  and  the 
Supervisory  Control  of  UCAVS  in  Tactical  C2: 
Effects  on  Performance  and  Workload 


Gregory  J.  Funke 

General  Dynamics  Advanced  Information  Systems 
Dayton  OH  45431 


April  M.  Bennett 
W.  Todd  Nelson 
Scott  M.  Galster 

Human  Effectiveness  Directorate 
Warfighter  Interface  Division 
Wright-Patterson  AFB  OH  45433-7022 


May  2007 

Interim  Report  for  May  2007  to  May  2007 


Approved  for  public  release; 
Distribution  is  unlimited. 


Air  Force  Research  Laboratory 
Human  Effectiveness  Directorate 
Warfighter  Interface  Division 
Collaborative  Interfaces  Branch 
Wright  Patterson  AFB  OH  45433 


NOTICE  AND  SIGNATURE  PAGE 


Using  Government  drawings,  specifications,  or  other  data  included  in  this  document  for 
any  purpose  other  than  Government  procurement  does  not  in  any  way  obligate  the  U.S. 
Government.  The  fact  that  the  Government  formulated  or  supplied  the  drawings, 
specifications,  or  other  data  does  not  license  the  holder  or  any  other  person  or  corporation; 
or  convey  any  rights  or  pennission  to  manufacture,  use,  or  sell  any  patented  invention  that 
may  relate  to  them. 

This  report  was  cleared  for  public  release  by  the  Air  Force  Research  Laboratory  Wright  Site 
Public  Affairs  Office  and  is  available  to  the  general  public,  including  foreign  nationals. 
Copies  may  be  obtained  from  the  Defense  Technical  Infonnation  Center  (DTIC) 
(http://www.dtic.mil). 


TECHNICAL  REVIEW  AND  APPROVAL 
AFRL-HE-WP-TP-2007-0013 


THIS  TECHNICAL  REPORT  HAS  BEEN  REVIEWED  AND  IS  APPROVED  FOR 

PUBLICATION. 


FOR  THE  DIRECTOR 


//signed// 

Scott  M.  Galster 

Experimental  Research  Psychologist 
Collaborative  Interfaces  Branch 


//signed// 

Daniel  G.  Goddard 

Chief,  Warfighter  Interfaces  Division 

Human  Effectiveness  Directorate 


This  report  is  published  in  the  interest  of  scientific  and  technical  information  exchange,  and  its 
publication  does  not  constitute  the  Government’s  approval  or  disapproval  of  its  ideas  or  findings. 


REPORT  DOCUMENTATION  PAGE 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  data  sources, 

gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection 

of  information,  including  suggestions  for  reducing  this  burden  to  Washington  Headquarters  Service,  Directorate  for  Information  Operations  and  Reports, 

1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202-4302,  and  to  the  Office  of  Management  and  Budget, 

Paperwork  Reduction  Project  (0704-0188)  Washington,  DC  20503. 

PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 

1.  REPORT  DATE  (DD-MM-YYYY) 

2.  REPORT  TYPE 

3.  DATES  COVERED  (From  -  To) 

18  May  2007 

Interim 

_ 

May  2007-  May  2007 

4.  TITLE  AND  SUBTITLE 

Collaboration  Technologies  and  the  Supervisory  Control  of  UCAVS  in 
Tactical  C2:  Effects  on  Performance  and  Workload 


5a.  CONTRACT  NUMBER 

FA8650-06-C-6755 


5b.  GRANT  NUMBER 


5c.  PROGRAM  ELEMENT  NUMBER 

62202F 


6.  AUTHOR(S) 

Gregory  J.  Funke* 
April  M.  Bennett** 
W.  Todd  Nelson** 
Scott  M.  Galster** 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

General  Dynamics  Advanced  Information  Systems* 
Dayton  OH  45431 


5d.  PROJECT  NUMBER 

7184 


5e.  TASK  NUMBER 

08 


5f.  WORK  UNIT  NUMBER 

72 


8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Air  Force  Materiel  Command** 

Air  Force  Research  Laboratory 
Human  Effectiveness  Directorate 
Warfighter  Interface  Division 
Collaborative  Interfaces  Branch 
Wright  Patterson  AFB  OH  45433-7022 


12.  DISTRIBUTION  AVAILABILITY  STATEMENT 


10.  SPONSOR/MONITOR'S  ACRONYM(S) 

AFRL/HECP 


11.  SPONSORING/MONITORING 
AGENCY  REPORT  NUMBER 

AFRL-HE-WP-TP-2007-001 3 


Approved  for  public  release;  distribution  is  unlimited. 


13.  SUPPLEMENTARY  NOTES 


AFRL/PA  Cleared  6/19/2007,  AFRL/WS-07-1446. 


14.  ABSTRACT 

The  present  study  was  an  initial  attempt  to  characterize  team  performance,  workload,  and  situational  awareness 
associeated  with  two  types  of  UCAV  control  schemes  coupled  with  several  collaboration  technologies.  Six  people 
participated  in  a  simulated  suppression  of  enemy  air  defense  (SEAD)  mission,  which  required  cooperation  between  all 
participants  in  order  to  meet  mission  objectives.  UCAVs  were  controlled  by  UCAV  operators  and  supervised  by  air  battle 
managers  (ABMs)  or  controlled  directly  by  ABMs.  Participants  could  communicate  verbally,  through  instant  messages, 
and  on  some  trials,  using  a  virtual  whiteboard.  Results  of  the  experiment  indicated  that  team  performance  was  negatively 
impacted  by  direct  UCAV  control  and  communication  using  the  virtual  whiteboard. _ 


15.  SUBJECT  TERMS 

Suppression  of  enemy  air  defense  (SEAD),  surface-to-air  missile  (SAM),  Unmanned  combat  aerial  defense  (UCAV),  Air 
battle  managers  (ABMs) _ 


16.  SECURITY  CLASSIFICATION  OF: 


Unclassified 


a.  REPORT 

U 


17.  LIMITATION  OF  118.  NUMBER 
ABSTRACT  I  OF  PAGES 


b.  ABSTRACT 

U 


c.  THIS  PAGE 

U 


19a.  NAME  OF  RESPONSIBLE  PERSON 

Scott  M.  Galster 


19b.  TELEPONE  NUMBER  (Include  area  code) 


Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI-Std  Z39-18 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


11 


COLLABORATION  TECHNOLOGIES  AND  THE  SUPERVISORY  CONTROL  OF  UCAVS  IN  TACTICAL  C2:  EFFECTS 

ON  PERFORMANCE  AND  WORKLOAD 


Gregory  J,  Funke 

General  Dynamics  Advanced  Information  Systems 
Dayton,  OH,  USA 

April  M.  Bennett,  W,  Todd  Nelson,  Scott  M.  Galster 
Air  Force  Research  Laboratory 
Wright-Patterson  Air  Force  Base,  OH,  USA 

The  present  study  was  an  initial  attempt  to  characterize  team  performance,  workload,  and  situational 
awareness  associated  with  two  types  of  UCAV  control  schemes  coupled  with  several  collaboration 
technologies.  Six  people  participated  in  a  simulated  suppression  of  enemy  air  defense  (SEAD)  mission, 
which  required  cooperation  between  all  participants  in  order  to  meet  mission  objectives.  UCAVs  were 
controlled  by  UCAV  operators  and  supervised  by  air  battle  managers  (ABMs)  or  controlled  directly  by 
ABMs.  Participants  could  communicate  verbally,  through  instant  messages,  and  on  some  trials,  using  a 
virtual  whiteboard.  Results  of  the  experiment  indicated  that  team  performance  was  negatively  impacted  by 
direct  UCAV  control  and  communication  using  the  virtual  whiteboard.  Overall,  these  results  suggest  that 
direct  UCAV  control  may  have  subtle,  yet  substantial,  negative  impact  on  several  aspects  of  team 
performance  and  that  efficient  use  of  collaboration  technologies  in  temporally  demanding  environments 
may  require  operators  to  first  develop  effective  communication  strategies. 


INTRODUCTION 

The  E-3  Sentry,  an  Airborne  Warning  and  Control  System 
(AW ACS)  aircraft,  is  highly  regarded  as  the  premier  air  battle 
management  command  and  control  (C2)  platform  and 
provides  the  U.S.  military  with  the  surveillance,  command, 
control,  and  communications  needed  for  effective  battle 
management,  and  its  capabilities  continue  to  evolve. 

Concurrently  with  the  AWACS’s  continued  development, 
considerable  military  interest  has  been  expressed  in  the 
development  of  unmanned  combat  aerial  vehicles  (UCAVs) 
for  use  in  suppression  of  enemy  air  defense  (SEAD)  missions 
(Worch  et  al.,  1996).  SEAD  missions  require  air  assets  to 
suppress  or  destroy  enemy  air  defenses,  usually  by  compelling 
anti-aircraft  or  surface-to-air  missile  (SAM)  sites  to  activate 
their  radars,  and  subsequently  firing  anti-radiation  missiles  at 
the  activated  sites  (e.g.,  Li  et  al.,  2002).  Currently,  SEAD 
missions  are  some  of  the  most  dangerous  and  demanding  that 
a  pilot  can  undertake  (Nelson  &  Bolia,  2006),  making  them 
ideal  for  UCAVs. 

UCAV  Control 

At  present,  air  battle  managers  (ABMs)  aboard  AW  ACS 
are  responsible  for  the  tactical  control  of  air  interdiction  and 
deep  strike,  both  of  which  involve  the  coordination  of  fighter 
cover,  strike  assets,  airborne  refueling,  and  SEAD  (e.g.,  Kopp, 
2002).  Within  the  framework  of  current  and  near-term 
concepts  of  operations,  the  inclusion  of  UCAV  assets  in 
SEAD  missions  will  require  ABMs  to  expand  their  duties  to 
include  management  of  teams  of  UCAV  operators.  However, 
this  command  structure  could  place  increased  workload 
demands  on  ABMs  as  they  must  dynamically  communicate 
with  and  coordinate  SEAD  mission  assets  in  order  to  meet 
mission  objectives. 


Alternatively,  some  analysts  have  advocated  direct 
control  of  UCAVs  by  ABMs,  thereby  expanding  the  ABMs' 
responsibilities  while  simultaneously  eliminating  the  need  for 
additional  personnel  (e.g.,  Kopp,  2002).  It  has  been  suggested 
that  direct  ABM  control  of  UCAVs  may  yield  several  tactical 
benefits  including  reduced  deployment  costs,  reduced  sensor- 
to-shooter  time,  and  more  seamless  integration  of  manned  and 
unmanned  SEAD  assets  (Nelson  &  Bolia,  2006). 

However,  direct  UCAV  control  by  ABMs  has  been 
criticized  on  several  grounds,  including  the  possibility  that 
direct  control  may  simply  change  ABMs’  workload  from 
communication  with  UCAV  operators  to  workload  associated 
with  monitoring  UCAV  operations,  a  change  which  may 
engender  unintended  consequences  (Parasuraman  &  Riley, 
1997).  Specifically,  this  shift  brings  into  question  the  impact 
of  direct  control  on  ABMs’  mental  workload,  situational 
awareness  (SA),  and  decision  and  mission  effectiveness,  and 
what,  if  any,  countermeasures  may  be  devised  to  address  these 
issues  (see  Nelson  &  Bolia.,  2006  for  a  comprehensive 
review). 

Collaboration  Technologies 

Recent  trends  in  military  acquisition  have  emphasized  the 
desire  to  introduce  collaboration  technologies  into  C2 
environments  (Kaufman,  2005).  Proponents  of  network- 
centric  operations  have  argued  that  these  technologies  might 
engender  increased  SA  and  task  flexibility  (Alberts  &  Hayes, 
2003).  These  technologies  could  potentially  mitigate  ABMs’ 
expected  workload  increases  associated  with  UCAV 
operations. 

However,  research  investigating  the  influence  of 
collaboration  technologies  on  team  performance  has  generally 
indicated  they  exert  a  substantial  negative  effect  on  team 
performance.  For  example,  Bordia  (1997)  concluded  that 
teams  restricted  to  collaboration  technologies  made  poorer 
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decisions,  measured  both  objectively  (e.g.,  meeting  task  goals) 
and  subjectively  (e.g.,  quality  of  solutions),  and  took  more 
time  to  reach  a  decision.  Moreover,  team  members 
experienced  less  satisfaction  with  team  processes. 

Conversely,  recent  research  conducted  by  Funke,  Galster, 
Nelson,  and  Dukes  (2006)  found  no  adverse  effects  of  one 
collaboration  technology,  instant  messaging  (IM),  on  team 
performance  in  a  simulated  C2  task.  Participants  restricted  to 
IM  communication  performed  at  a  rate  comparable  to 
participants  who  were  not  so  restricted. 

The  current  study  represents  an  initial,  exploratory 
attempt  to  characterize  team  performance,  workload,  and  SA 
associated  with  several  UCAV  control  schemes  and 
collaboration  technologies. 

METHOD 

Participants 

Six  people  (5  men,  1  woman)  between  the  ages  of  19  and 
36  (M  =  23.5,  SE  =  2.59)  served  as  paid  participants  in  this 
study.  Participants  all  had  prior  experience  with  the  DDD 
interface  used  in  this  study;  however,  none  of  them  had  prior 
experience  with  the  current  scenario.  Participants  were 
required  to  have  normal  or  corrected-to-normal  vision  and 
normal  color  vision. 

Experimental  Design 

A  2  x  2  within-subjects  design  was  employed,  with  two 
UCAV-control  conditions  (direct,  supervisory)  combined 
factorially  with  two  levels  of  collaboration  condition 
(standard,  whiteboard)  and  six  levels  of  team  position  (ABM- 
sweep,  ABM-UCAV,  sweep-operator- 1,  sweep-operator-2, 
bomber-operator,  UCAV-operator). 

UCAV  control  and  team  position  were  block  factors,  and 
collaboration  condition  was  randomized  within  each  block. 
Participants  completed  two  trials  in  each  condition,  for  a  total 
of  48  trials. 

Measures  of  individual  and  team  performance  included 
the  number  of  enemy  targets  destroyed,  the  number  of  team 
assets  killed,  and  frequency  and  type  of  team  communication. 
Subjective  measures  included  the  NASA  Task  Load  Index 
(TLX;  Hart  &  Staveland,  1988)  and  the  3-D  Situational 
Awareness  Rating  Technique  (SART;  Taylor,  1990). 

Apparatus 

The  simulated  environment  utilized  was  Aptima,  Inc.’s 
Distributed  Dynamic  Decision-making  (DDD)  simulation,  a 
team-in-the-loop  simulated  environment  which  has  previously 
been  employed  in  a  variety  of  military  and  civilian  research 
projects.  The  simulation  required  16  networked  PCs 
communicating  under  TCP/IP  protocol.  Each  workstation  was 
equipped  with  a  keyboard,  mouse,  and  two  monitors.  The 
DDD  display  was  projected  on  a  17-inch  ViewSonic  VP720b 
LCD  monitor  and  the  Windows  operating  environment  was 
projected  on  an  18-inch  NEC  MultiSync  1800  LCD  monitor. 
A  Synergy  KM  switch,  model  1.3.1,  allowed  participants  to 
interact  with  desktop  objects  on  each  monitor  with  a  single 
mouse.  Verbal  communication,  participant  behavior,  and 


screen  captures  were  recorded  using  Nuance  SDK  (version 
8.5)  and  MORAE  (version  1.1.1)  software. 

The  whiteboard  collaboration  condition  utilized  the 
Dynamic  Real-time  Animated  Whiteboard  (DRAW) 
application.  The  DRAW  tool  is  an  experimental,  virtual 
whiteboard  which  enables  users  to  annotate  and  send  images 
of  a  battle-space  to  each  other. 

Simulation 

The  DDD  was  used  to  create  a  simulated  SEAD  mission 
scenario,  in  which  participants  were  tasked  with  following  a 
specified  flight  path,  conveyed  through  a  tactical  display,  in 
order  to  meet  SEAD  mission  goals.  The  objective  of  each 
mission  trial  was  to  destroy  as  many  enemy  targets  as  possible 
while  simultaneously  protecting  team  assets. 

The  ABMs  coordinated  the  actions  of  other  team 
members  in  order  to  meet  objectives.  ABM-sweep  directed  the 
movement  and  target  acquisition  of  both  sweep-operators.  The 
responsibilities  of  ABM-UCAV  were  contingent  upon  the 
UCAV-control  condition  of  each  trial.  In  the  supervisory 
UCAV-control  condition,  ABM-UCAV  directed  the  bomber- 
and  the  UCAV-operator.  In  the  direct  UCAV-control 
condition,  the  role  of  UCAV-operator  was  subsumed  by 
ABM-UCAV,  who  then  manually  controlled  UCAV  assets  in 
addition  to  directing  the  bomber-operator. 

At  the  start  of  each  mission  trial,  each  operator  controlled 
four  assets  of  the  same  type.  Each  asset  had  a  limited  amount 
of  ammunition  and  fuel.  As  these  resources  were  depleted, 
assets  would  have  to  return  to  a  ‘home-base’  to  be  restocked 
and  refueled.  Assets  that  ran  out  of  ammunition  were  unable  to 
attack,  and  assets  that  ran  out  of  fuel  ‘died’  and  were  removed 
from  the  simulation.  Initial  fuel  levels  of  each  asset  were 
randomized  between  three  to  nine  minutes  of  fuel  to  ensure 
each  asset  would  require  refueling.  After  refueling,  fighter 
and  bomber  assets  were  equipped  with  seven  minutes  of  fuel, 
and  UCAV  assets  were  equipped  with  nine  minutes  of  fuel. 

Additional  assets  of  each  type  were  available  to  operators 
from  home-base  to  replace  units  that  were  killed  or  that 
needed  restocking  or  refueling.  Participants  were  instructed  to 
maintain  four  assets  of  the  appropriate  type  in  the  simulation 
at  all  times,  and  to  launch  additional  assets  from  home-base  to 
replace  assets  as  needed.  However,  operators  had  to  receive 
orders  from  their  respective  ABM  to  launch  from  or  return 
assets  to  home-base. 

To  further  simulate  the  role  of  ABMs  in  a  combat 
situation,  the  tactical  displays  of  UCAV-,  bomber-,  and 
sweep-operators  conveyed  limited  information  concerning 
enemy  units  within  the  simulated  battle-space.  The  ABMs’ 
displays,  on  the  other  hand,  were  not  limited  in  this  fashion, 
giving  them  a  more  complete  understanding  of  the  battle- 
space. 

The  capabilities  and  assets  of  each  operator  were 
specialized  for  their  role  within  the  SEAD  mission. 
Specifically,  the  UCAV-operator  was  responsible  for 
eliminating  enemy  surface-to-air-missile  (SAM)  sites.  The 
bomber-operator  was  responsible  for  destroying  strategic 
enemy  ground  targets.  The  two  sweep-operators  were 
collectively  responsible  for  protecting  assets  from  enemy 
fighter  aircraft.  Each  unique  role  was  designed  such  that  each 


2 


position’s  contributions  could  not  be  replicated  by  those  of 
another  position  (e.g.,  a  UCAV  could  destroy  a  SAM  site,  but 
a  bomber  could  not).  This  division  of  simulated  capabilities 
resulted  in  a  situation  in  which  success  or  failure  of  each 
mission  trial  was  dependent  upon  the  contributions  of  all  team 
members. 

Each  ten-minute  mission  trial  featured  36  SAM  sites  and 
24  bridges  randomly  distributed  around  the  flight  path.  Thirty- 
six  enemy  MiGs  were  programmed  to  enter  the  scenario  at 
random  intervals  during  each  mission  trial.  SAM  sites  and 
bridges  were  stationary,  while  MiGs  were  mobile.  Both  SAMs 
and  MiGs  could  ‘kill’  team  assets,  while  bridges  were  passive 
entities  which  did  not  threaten  team  assets. 

Procedure 

Training.  Participants  received  42  hours  of  instruction  and 
practice  with  the  SEAD  mission  task,  distributed  across  ten 
days.  Participants  first  completed  a  computer-based  training 
module,  which  instructed  them  on  the  nature  and 
responsibilities  of  the  task  as  well  as  familiarized  them  with 
the  controls  of  the  DDD  interface.  Participants  then  received 
more  role  specific  instruction  for  each  team  position,  and 
direct  training  with  the  DDD  interface. 

Participants  also  received  instruction  on  the  use  of  the 
radio  communication  equipment,  instant  messaging,  and 
virtual  whiteboard  employed  in  the  experiment.  To  discourage 
verbal  communication  through  other  means,  a  recording  of 
pink  noise  was  played  in  the  lab  during  each  mission  trial. 
Participants  also  received  training  on  completion  of  the 
NASA-TLX  and  3-D  SART. 

Participants  practiced  until  they  could  demonstrate 
proficiency  at  all  positions.  Although  each  position  differed 
slightly,  in  general,  this  entailed  achieving  75%  of  mission 
goals  in  a  single  trial.  Immediately  following  practice  trials, 
participants  were  debriefed  about  their  performance  and  about 
aspects  of  their  performance  that  needed  to  be  addressed  in 
order  to  demonstrate  proficiency. 

Experimental  data  collection.  The  experimental  data 
collection  phase  of  the  experiment  began  immediately 
following  completion  of  training.  Data  collection  required  15 
hours,  distributed  over  3  days.  Each  participant  was  randomly 
assigned  to  an  order  of  team  positions.  Participants  then 
completed  eight  trials  in  that  team  configuration,  before 
rotating  to  their  next  team  position.  This  process  was 
duplicated  six  times,  until  each  participant  had  completed 
eight  trials  in  each  team  position.  During  each  eight-trial 
block,  participants  completed  four  trials  in  each  collaboration 
condition. 

After  completing  all  experimental  trials,  participants 
completed  a  debriefing  questionnaire.  This  measure  was 
designed  to  elicit  participants’  impressions  of  the  SEAD 
mission,  the  DDD  software,  and  the  collaboration  tools 
employed  in  the  experiment. 

RESULTS 

To  examine  potential  differences  in  team  response  to  the 
experimentally  manipulated  factors,  an  ABM-team  factor  with 
two  levels  (sweep-team,  UCAV-team)  was  conceived.  Sweep- 


team  consisted  of  ABM-sweep  and  the  two  sweep  operators. 
UCAV-team  consisted  of  ABM-UCAV  and  the  bomber-  and 
UCAV-operators. 

Team  Performance 

Enemy  targets  destroyed.  To  determine  the  effects  of  the 
experimental  manipulations  on  the  number  of  enemy  targets 
destroyed,  the  percentage  of  ABM-team  targets  destroyed  in 
each  trial  was  computed.  The  data  for  this  factor  were  tested 
for  statistical  significance  by  means  of  a  2  (control  condition) 
x  2  (collaboration  technology)  x  2  (ABM-team)  repeated 
measures  analysis  of  variance  (ANOVA).  A  statistically 
significant  main  effect  of  ABM-team  was  detected,  F  (1,5)  - 
22.19,  /?<.05.  The  mean  percentage  of  ABM-team  targets 
destroyed  was  significantly  higher  for  the  ABM-sweep  team 
(, M  =  99.36,  SE  =  .23)  compared  to  the  ABM-UCAV  team  (M 
=  82.78,  SE=  3.58). 

Assets  killed.  The  effects  of  the  experimentally 
manipulated  factors  on  the  numbers  of  each  type  of  team  asset 
killed  (fighters,  bombers,  UCAVs)  were  tested  by  separate  2 
(control  condition)  x  2  (collaboration  technology)  repeated 
measures  ANOVAs.  For  the  number  of  fighter  assets  killed, 
all  sources  of  variance  in  the  analysis  were  not  significant  [p  > 
.05).  However,  a  trend  within  the  data  suggested  that  more 
fighter  assets  were  killed  in  the  direct  control  condition  (M  = 
2.71,  SE  =  .20)  compared  to  supervisory  control  condition  (M 
=  1.92,  SE  =  .27)  (p  <  .10). 

For  the  number  of  bomber  assets  killed,  a  statistically 
significant  main  effect  of  collaboration  condition  was 
detected,  F  (1,  5)  =  12.00,  p  <  .05.  Fewer  bomber  assets  were 
killed  in  the  standard  collaboration  condition  ( M  =  .71,  SE  = 
.14)  compared  to  DRAW  condition  (Af  s=  1.21,  SE  =  .17). 

For  the  number  of  UCAV  assets  killed,  a  statistically 
significant  UCAV-control  x  collaboration  condition 
interaction  was  detected,  F  (1,  5)  =  7.49,  p  <  .05.  As  is 
illustrated  in  Figure  1,  the  greatest  number  of  UCAV  losses 
occurred  in  the  direct  UCAV-control  DRAW  collaboration 
condition. 


UCAV-Control 

Figure  1.  Mean  number  of  UCAV  assets  killed  as  a  function  of  UCAV- 
control  and  collaboration  conditions.  Error  bars  are  standard  errors. 

To  further  explore  the  UCAV-control  x  collaboration 
interaction,  separate  repeated  measures  ANOVAs  were 
calculated  in  which  collaboration  condition  was  analyzed  for 
each  UCAV-control  condition.  For  direct  control,  a 
statistically  significant  main  effect  of  collaboration  condition 
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was  detected  (F  [1, 5]  =  14.31,  p  <  .05),  but  no  such  difference 
was  detected  for  the  supervisory  control  condition  (p  >  .05). 
Fewer  UCAV  assets  were  killed  under  direct  UCAV  control  in 
the  standard  collaboration  condition  (A/  -  .33,  SE  -  .17) 
compared  to  the  DRAW  condition  ( M  =  1.75,  SE  =  .44). 

Base  launches.  Both  ABM-teams  were  responsible  for 
launching  new  assets  from  home-base  to  replace  assets  that 
were  killed,  or  refueling.  Therefore,  the  optimal  number  of 
base  launches  in  any  trial  was  equal  to  the  number  of  assets 
returned  to  home  base  plus  the  number  of  assets  killed. 
However,  participants  did  not  always  perfectly  adhere  to  these 
instructions.  An  index  of  launch  inefficiency  (LI)  can  be 
calculated  as  LI  =  observed  launches  -  optimal  launches.  This 
index  was  calculated  for  each  condition  across  trials  and 
compared  by  means  of  a  2  (control  condition)  x  2 
(collaboration  technology)  x  2  (ABM-team)  repeated 
measures  ANOVA.  For  launch  inefficiency,  a  statistically 
significant  main  effect  of  ABM-team  (F  [1,  5]  =  44.93,  p  < 
.05)  and  a  statistically  significant  UCAV-control  x  ABM-team 
interaction  (F  [1,  5]  =  7.79,  p  <  .05)  were  detected.  As  is 
illustrated  in  Figure  2,  the  sweep-team  maintained  fewer  assets 
in  the  simulation  than  the  UCAV-team. 

To  further  explore  the  UCAV-control  x  ABM-team 
interaction,  separate  repeated  measures  ANOVAs  were 
calculated  in  which  UCAV-control  condition  was  analyzed  for 
each  ABM-team.  For  the  sweep-team,  a  statistically 
significant  main  effect  of  UCAV-control  condition  was 
detected  (F  (1, 5]  =  13.89,  p  <  .05),  but  no  such  difference  was 
detected  for  the  UCAV-team  ip  >  .05).  The  sweep-team 
maintained  a  greater  number  of  assets  in  the  simulation  in  the 
direct  control  condition  (M  =  -2.17,  SE  =  .51)  compared  to  the 
supervisory  control  condition  ( M  =  -2.79,  SE  =  .48). 


Figure  2.  Mean  launch  inefficiency  (absolute  values)  as  a  function  of 
UCAV-control  and  ABM-team  conditions.  Error  bars  are  standard 
errors. 

Subjective  Workload  and  Situational  Awareness 

Workload.  A  2  (control  condition)  x  2  (collaboration 
technology)  x  6  (TLX  subscale)  repeated  measures  ANOVA 
indicated  a  statistically  significant  main  effect  of  TLX 
subscale,  F  (1.29,  6.45)  =  14.60,  p  <  .05.  In  this  analysis. 
Box’s  epsilon  was  employed  to  correct  for  violations  of  the 
sphericity  assumption  (Maxwell  &  Delaney,  2004).  A 
subsequent  post  hoc  Tukey’s  HSD  of  TLX  subscale  revealed 
no  significant  differences  between  subscales  (all  p  >  .05). 


Participants’  ratings  of  workload  were  not  influenced  by  the 
experimentally  manipulated  factors. 

Situational  awareness.  A  2  (control  condition)  x  2 
(collaboration  technology)  repeated  measures  ANOVA 
detected  no  significant  sources  of  variance  in  the  analysis  (all 
p  >  .05).  The  experimentally  manipulated  factors  did  not 
influence  participants’  ratings  of  situational  awareness. 

Team  Communication 

To  determine  the  effects  of  the  experimental 
manipulations  on  team  communications,  the  mean  number  of 
verbal,  IM,  and  whiteboard  messages  sent  was  calculated  for 
each  condition  across  trials.  However,  examination  of  the 
mean  number  of  IMs  sent  indicated  that  participants  did  not 
use  this  tool  for  collaboration,  preventing  further  analysis  of 
its  impact  on  team  performance. 

Verbal  communication.  A  2  (control  condition)  x  2 
(collaboration  technology)  x  2  (ABM-team)  repeated 
measures  ANOVA  revealed  statistically  significant  main 
effects  of  UCAV-control  condition  (F  (1,  5]  =  20.33,  p  <  .05), 
ABM-team  (F  [1,  5]  =  11.31,  p  <  .05),  and  a  statistically 
significant  UCAV-control  x  ABM-team  interaction  (F  (1,  5]  = 
35.10,  p  <  .05).  As  illustrated  in  Figure  3,  participants  sent 
more  verbal  communications  in  the  direct  UCAV-control 
condition  compared  to  the  supervisory  condition,  and  the 
sweep-team  sent  more  verbal  messages  than  the  UCAV-team 
across  conditions. 
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Figure  3.  Mean  number  of  verbal  messages  sent  as  a  function  of 
UCAV-control  and  ABM-team  conditions.  Error  bars  are  standard 
errors. 

To  further  explore  the  UCAV-control  x  ABM-team 
interaction,  separate  repeated  measures  ANOVAs  were 
calculated  in  which  UCAV-control  condition  was  analyzed  for 
each  ABM-team.  For  the  UCAV-team,  a  statistically 
significant  main  effect  of  UCAV-control  condition  was 
detected  (F  [1,  5]  =  27.94,  p  <  .05),  but  no  such  difference  was 
detected  for  the  sweep-team  ip  >  .05).  The  UCAV-team  sent 
approximately  a  fourth  fewer  verbal  messages  in  the  direct 
control  condition  (Af  =  23.13,  SE  =  3.48)  compared  to  the 
supervisory  control  condition  (Af  =  88.71,  SE  =  1 1.65). 

Whiteboard  communication.  A  2  (control  condition)  x  2 
(ABM-team)  repeated  measures  ANOVA  detected  a 
statistically  significant  UCAV-control  x  ABM-team 
interaction,  F  (1 , 5)  =  8. 1 1 ,  p  <  .05. 
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To  further  explore  the  UCAV-control  x  ABM-team 
interaction,  separate  repeated  measures  ANOVAs  were 
calculated  in  which  UCAV-control  condition  was  analyzed  for 
each  ABM-team.  For  the  UCAV-team,  a  statistically 
significant  main  effect  of  UCAV-control  condition  was 
detected  (F  [1,  5]  =  7.98,  p  <  .05),  but  no  such  difference  was 
detected  for  the  sweep-team  (p  >  .05).  The  UCAV-team  sent 
approximately  a  third  fewer  whiteboard  images  in  the  direct 
control  condition  (M  =  5.83,  SE  =  1.25)  compared  to  the 
supervisory  control  condition  ( M  =  17.33,  SE  =  5.04). 

Collaboration  preference.  Items  on  the  post-experimental 
debriefing  questionnaire  asked  participants  to  separately 
estimate  the  usefulness  of  and  satisfaction  from 
communicating  using  the  3  different  tools  available  in  the 
experiment  (verbal,  IM,  DRAW).  Participants  were  asked  to 
rate  the  three  tools  on  a  scale  of  0  to  100,  where  0  was  low 
usefulness  or  satisfaction,  and  100  was  high  usefulness  or 
satisfaction.  Participants  were  also  asked  to  justify  the  ratings 
they  gave  each  tool.  Mean  participant  ratings  for  each  tool  are 
presented  in  Table  1. 

Table  1.  Mean  participant  ratings  of  the  usefulness  and  satisfaction  of 
each  collaboration  tool.  Values  in  parentheses  are  standard  errors. 


Tool 

Usefulness 

Satisfaction 

Verbal 

98.33  (1.67) 

95.83  (4.17) 

IM 

3.33  (1.67) 

2.50  (1.71) 

DRAW 

76.67  (6.41) 

80.00  (7.30) 

As  indicated  in  Table  1,  participants  rated  verbal 
communication  to  be  the  most  useful  and  most  satisfying  tool 
for  collaboration,  with  the  DRAW  tool  as  a  close  second,  and 
ratings  of  IM  as  near  zero.  For  verbal  communication,  the 
most  common  explanations  for  the  above  ratings  were  that 
verbal  was  the  easiest  (4  participants)  and  fastest  (3 
participants)  collaboration  method  available.  The  most  typical 
responses  to  the  DRAW  tool  were  that  it  was  useful  for 
communicating  the  spatial  locations  of  enemy  targets  (6 
participants)  but  that  it  was  generally  too  slow  for  most 
communication  (2  participants).  Overwhelmingly,  participants 
argued  that  JM  was  impractically  slow  for  collaboration,  given 
the  high  temporal  pressures  of  the  SEAD  mission  simulation 
(6  participants). 

DISCUSSION 

The  current  study  was  an  initial,  exploratory  attempt  to 
characterize  team  performance,  workload,  and  situational 
awareness  associated  with  direct  and  supervisory  control  of 
UCAVs  by  ABMs  coupled  with  several  collaboration 
technologies  in  a  SEAD  mission  setting. 

UCAV  Control 

Team  performance.  The  effects  of  UCAV-control 
condition  on  team  performance  were  complex,  and 
interestingly,  not  limited  to  the  ABM-UCAV/operator  team. 
Superficially,  these  effects  appear  disparate  and  inconsistent 
across  indices  of  team  performance.  When  viewed  holistically, 
however,  they  may  indicate  a  shift  by  the  ABM-UCAV  in  the 


direct  UCAV-control  condition  to  an  acceptance  of  riskier 
decisions  in  pursuit  of  task  goals. 

This  viewpoint  may  be  supported  by  the  increase  in 
fighter  asset  losses  and  the  decrease  in  launch  inefficiency  by 
the  sweep-team  under  direct  control  conditions.  Within  the 
simulation,  the  responsibilities  of  the  sweep-team  were  to 
protect  UCAV-team  assets  from  enemy  MiGs.  If  the  ABM- 
UCAV  adopted  a  riskier  decision  making  strategy  under  direct 
control  conditions,  this  may  have  led  to  the  acceptance  of 
more  MiG-vulnerable  UCAV-team  asset  deployments.  This 
posture  would  then  necessitate  fighter  asset  engagements 
under  less  ideal  circumstances  as  the  sweep-team  attempted  to 
meet  their  task  objectives,  resulting  in  increased  fighter  asset 
losses  and  more  efficient  sweep-team  launch  strategies  to 
replace  those  killed  assets.  Further  compounding  the  issue,  the 
shift  to  a  riskier  decision-making  strategy  is  not  supported  by 
a  concomitant  increase  in  the  number  of  enemy  targets 
destroyed  for  either  ABM.  Overall,  direct  control  of  UCAV 
assets  by  the  ABM-UCAV  resulted  in  subtle,  yet  substantial, 
negative  impact  on  team  performance. 

Workload  and  situational  awareness.  Participants’  ratings 
of  workload  and  situational  awareness  were  unchanged  by 
UCAV-control  condition.  The  lack  of  significant  workload 
change  under  direct  UCAV  control  is  of  note.  Three  potential 
explanations  may  account  for  this.  First,  it  may  be  that 
increased  workload  associated  with  direct  UCAV  control  may 
be  equally  offset  by  the  reduction  of  workload  related  to 
operator  supervision  (i.e.,  both  UCAV-control  conditions 
resulted  in  equal).  This  explanation,  however,  does  not 
adequately  explain  the  shifts  in  team  performance  observed 
under  direct  UCAV  control. 

Alternatively,  it  may  be  possible  that  participants  in  the 
role  of  ABM-UCAV  offset  additional  workload  associated 
with  direct  UCAV  control  by  accepting  riskier  task  strategies, 
as  discussed  previously.  As  noted  by  Singleton  ( 1989),  if  the 
workload  demands  of  a  task  exceed  operators’  abilities  to 
manage,  a  skilled  operator  may  adjust  their  performance 
strategies  to  compensate.  One  such  adjustment  may  be  to 
expend  less  effort  in  meeting  task  demands,  leading  to  a 
decrement  in  task  performance  (Wickens  &  Hollands,  2000). 

Finally,*  the  competition  and  game-related  nature  of  the 
SEAD  task  may  have  motivated  participation  in  ways  that 
diminished  the  negative  effects  of  direct  UCAV  control.lt  is 
possible  that  participants  in  the  direct  control  condition 
expended  more  effort  and  yet  reported  no  changes  in  workload 
and  situational  awareness  because  they  were  sufficiently 
engaged  with  the  task.  Support  for  this  viewpoint  comes  from 
previous  research  linking  task  engagement  to  subjective 
energy  and  performance  on  difficult,  attentionally  demanding 
tasks  (e.g.,  Matthews  &  Westerman,  1994).  In  other  words, 
participants  may  have  been  insulated  against  the  negative 
effects  of  direct  UCAV  control  on  workload  and  situational 
awareness  through  motivation  and  engagement  with  the  task. 

Collaboration  Technologies 

Team  performance .  Access  to  the  virtual  whiteboard  in 
the  current  experiment  resulted  in  poorer  team  performance,  as 
indexed  by  the  number  of  bomber  and  UCAV  assets  killed. 
This  may  potentially  be  attributed  to  participants’  need  to 
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switch  attention  between  the  DDD  and  DRAW  displays, 
perhaps  frequently,  to  utilize  the  information  provided  by  the 
whiteboard  image.  The  additional  time  required  to  shift 
attention  from  the  DDD  display,  acquire  and  process 
information  from  the  DRAW  image,  and  then  shift  attention 
back  may  have  been  sufficiently  distracting  to  result  in  the 
performance  decrements  observed. 

Participants  may  also  have  adopted  an  inappropriate 
display-sampling  strategy  focused  on  obtaining  information 
from  whiteboard  images,  resulting  in  insufficient  attention  to 
SEAD  tasks,  and  ultimately,  poorer  team  performance.  As 
noted  by  Wickens  and  Hollands  (2000),  individuals  have  a 
tendency  to  seek  more  information  than  they  can  process  and 
integrate  quickly.  Under  conditions  of  temporal  demand,  such 
as  those  of  many  C2  environments,  this  may  result  in  impaired 
decision  making  as  a  consequence  of  the  information-rich 
setting  (Wright,  1974). 

Interestingly,  participants’  ratings  of  the  DRAW  tool  were 
seemingly  insensitive  to  this  decrement,  as  it  was  rated  very 
highly  in  both  usefulness  and  satisfaction  by  all  participants. 
This  may  indicate  that  personnel  in  C2  environments  need 
substantial  training,  beyond  functional  literacy,  with 
whiteboard  software  to  develop  optimal  strategies  for 
sampling  and  interpreting  whiteboard  images  under 
temporally  demanding  conditions. 

Workload  and  situational  awareness.  Access  to  a  virtual 
whiteboard  in  this  experiment  did  not  affect  participants’ 
ratings  of  workload  and  situational  awareness.  It  may  be  that 
the  additional  expenditures  of  time  and  effort  associated  with 
sending,  receiving,  and  interpreting  whiteboard  images  offset 
any  positive  impact  those  images  may  have  had  on  workload 
and  situational  awareness.  This  viewpoint  is  particularly 
interesting  in  light  of  participants’  comments  about  the  utility 
of  communicating  the  spatial  locations  of  enemy  targets  using 
whiteboard  images.  Participants  may  have  overestimated  the 
utility  of  a  virtual  whiteboard  for  communicating  spatial 
information,  but  misperceive  the  associated  costs  to  team 
performance. 

Overall,  the  results  raise  concerns  over  utilizing  a 
whiteboard  for  collaboration  in  time-sensitive  situations,  at 
least  as  it  was  utilized  in  this  experiment.  If  extensive  training 
with  the  tool  is  necessary  to  overcome  the  performance 
impairments  observed,  then  a  more  efficient  solution  may  be 
to  limit  its  use  to  collaborative  environments  that  are  not 
temporally  demanding.  Allowing  personnel  to  communicate 
using  this  innovative  medium  under  such  conditions  may 
facilitate  group  performance  and  group  satisfaction  in 
distributed  teams  without  the  associated  performance  deficits 
observed  in  this  experiment. 

Conclusions 

As  an  initial,  exploratory  attempt  to  characterize  team 
performance  associated  with  several  UCAV  control  schemes 
and  collaboration  technologies  in  a  SEAD  mission  setting,  the 
current  experiment  was  relatively  successful.  The  results  of 
the  experiment  raise  substantive  questions  about  the  ability  of 
ABMs  to  successfully  carry  out  SEAD  mission  objectives 
under  direct  UCAV  control,  and  about  the  effectiveness  of 


whiteboard  communications  in  C2  environments.  Both  direct 
UCAV  control  and  the  collaborative  technologies  investigated 
in  this  experiment  may  exert  widespread  negative  effects  on 
team  performance  in  a  complex  and  unanticipated  manner. 

Overall,  the  current  experiment  offers  some  insight  into 
the  multifaceted  human  factors  concerns  that  will  impact 
future  SEAD  mission  operations,  while  simultaneously 
provoking  further  questions  about  such  operations.  Addressing 
these  issues  is  necessary  to  ensure  successful  future  operations 
in  command  and  control  environments. 
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